feat: Support RowKind#70
Open
ggjh-159 wants to merge 1 commit into
Open
Conversation
8 tasks
ggjh-159
marked this pull request as draft
July 13, 2026 06:59
ggjh-159
marked this pull request as ready for review
July 13, 2026 07:49
ggjh-159
force-pushed
the
feat/support-rowkind
branch
from
July 16, 2026 09:20
c1e1867 to
f62e76b
Compare
ggjh-159
force-pushed
the
feat/support-rowkind
branch
from
July 16, 2026 09:30
f62e76b to
8692f43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of this change?
Flink RowData carries a RowKind header (+I / -U / +U / -D, byte ordinals 0/1/2/3) that distinguishes changelog records. Today the RowKind signal is dropped at the Flink→velox boundary, so every row exiting the stateful pipeline is +I regardless of the original changelog marker. This is the velox-side foundation for threading RowKind through the stateful operator chain.
What changes are proposed in this pull request?
RowKind lives on StreamRecord directly — no new wrapper class. The merged RowVector convention (trailing $row_kind TINYINT column, constant
kRowKindColumnNamein velox/experimental/stateful/RowKind.h) is the wire format across the JNI boundary; it is split/recombined at the StreamRecord layer.rowKind_: SimpleVectorPtr<int8_t>.nullptrmeans append-only.appendOnly()returns!rowKind_.toMergedRowVector(alwaysAppendRowKind = false)returns N-column RowVector (appendOnly) or (N+1)-column RowVector with trailing $row_kind TINYINT (changelog). WhenalwaysAppendRowKindis true, appendOnly records get a ConstantVector<int8_t>(INSERT) trailing column — used by callers that need a fixed N+1 schema.StreamRecord::create(nodeId, merged)splits a merged RowVector back into record + rowKind. A trailing ConstantVector<int8_t>(INSERT) is normalized back to appendOnly (rowKind = nullptr) so sources that always emit the column don't force every downstream operator onto the changelog path.StreamRecord::createis used to interpret whatever the source emits — a RowVector with a trailing $row_kind column yields a per-row RowKind StreamRecord; a plain RowVector yields an appendOnly StreamRecord. No assumption that "sources only emit INSERT" — CDC-style sources (Pulsar/Kafka CDC, ORC tables with metadata columns) can emit a $row_kind column directly.Related design + phased plan: support-rowkind-column-issue
Test
New C++ unit test
velox_stateful_changelog_row_vector_test